# DeepSeek deployment

***Copyright © Quectel Wireless Solutions Co., Ltd. 2026. All rights reserved.***

---

# DeepSeek Overview

DeepSeek is a high-performance open-source large language model developed by a domestic team, with strong natural language understanding, code generation, and logical reasoning capabilities. This document demonstrates how to deploy the DeepSeek-R1-Distill-Qwen-1.5B distilled model on the NPU of the Quectel PI M2 (RK3576) using the RKLLM (Rockchip LLM) toolchain. The full DeepSeek-V3/R1 (671B parameters) cannot run on embedded boards; this solution deploys the official distilled small model, retaining R1's reasoning (thinking) capability, suitable for edge-side Q&A and reasoning scenarios.

# Deployment Solution

| **Item** | **Detail** |
| --- | --- |
| SDK | RKLLM SDK v1.2.1 (built into external/rknn-llm/) |
| Model | DeepSeek-R1-Distill-Qwen-1.5B, W4A16 quantized |
| Inference hardware | RK3576 NPU (6 TOPS), single core |
| Board system | Debian 13 (trixie), 4GB RAM |

# Overall Architecture

```plaintext
┌────────────────────────── PC Side (x86_64) ──────────────────────────┐
│ 1. rkllm-toolkit (Python)                                          │
│ 2. Download original model from HuggingFace                        │
│ 3. Generate quantization calibration data (data_quant.json)        │
│ 4. Model conversion + quantization → .rkllm file                   │
│ 5. Cross-compile on-board demo (llm_demo)                          │
└──────────────────────────────────┬─────────────────────────────────┘
                                   │ adb push
┌────────────────────────── RK3576 Board ────────────────────────────┐
│ 6. librkllmrt.so (on-board runtime, aarch64 prebuilt)              │
│ 7. llm_demo loads .rkllm model → NPU inference                     │
└───────────────────────────────────────────────────────────────────┘
```

# Environment Preparation

## Hardware and Connection

| **Item** | **Description** |
| --- | --- |
| Development board | Quectel PI M2 (RK3576) |
| Memory | 4GB LPDDR4X |
| NPU | 6 TOPS, max frequency 950MHz |
| Connection | USB ADB (board adbd enabled, root) |
| PC system | Ubuntu 22.04, Python 3.10 |

## SDK Built-in Resources (No Extra Download)

```plaintext
external/rknn-llm/
├── rkllm-toolkit/                 # PC-side model conversion tool (whl included)
├── rkllm-runtime/                 # On-board runtime
│   └── Linux/librkllm_api/
│       ├── aarch64/librkllmrt.so  # aarch64 prebuilt runtime library
│       └── include/rkllm.h        # C/C++ API header
├── examples/
│   └── DeepSeek-R1-Distill-Qwen-1.5B_Demo/
│       ├── export/                # Model conversion scripts
│       └── deploy/                # On-board demo
├── scripts/fix_freq_rk3576.sh     # NPU/CPU frequency lock script
└── doc/                           # RKLLM SDK documentation PDF
```

# Detailed Setup Steps

## Install rkllm-toolkit (PC Side)

```bash
# Create venv, inherit system-installed torch/numpy
python3 -m venv --system-site-packages /home/q/rkllm-venv

# Install only rkllm-toolkit itself (no dependency pull)
/home/q/rkllm-venv/bin/pip install --no-deps \
  <SDK>/external/rknn-llm/rkllm-toolkit/rkllm_toolkit-1.2.1-cp310-cp310-linux_x86_64.whl

# Install missing small dependencies (use domestic mirror)
/home/q/rkllm-venv/bin/pip install -i https://mirrors.aliyun.com/pypi/simple/ \
  colorlog transformers==4.51.3 sentencepiece tiktoken einops safetensors \
  tqdm accelerate protobuf tabulate jinja2 jsonlines flatbuffers jsonschema datasets

# Verify
/home/q/rkllm-venv/bin/python -c "from rkllm.api import RKLLM; print('OK')"
```

**Gotcha:** The wheel metadata hardcodes versions like torch==2.3.0. A direct pip install of the whl will download the full dependency set (2GB+). Use --no-deps with the system torch to avoid this.

## Download the Original Model

Download from the HuggingFace mirror (hf-mirror is fast domestically):

```bash
mkdir -p /home/q/deepseek-1.5b && cd /home/q/deepseek-1.5b
for f in model.safetensors config.json generation_config.json tokenizer.json tokenizer_config.json; do
  wget -c "https://hf-mirror.com/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B/resolve/main/$f" -O "$f"
done
```

Model files: model.safetensors (~3.4GB) + 4 config files.

## Generate Quantization Calibration Data

```bash
cd /home/q/deepseek-1.5b
/home/q/rkllm-venv/bin/python \
  <SDK>/external/rknn-llm/examples/DeepSeek-R1-Distill-Qwen-1.5B_Demo/export/generate_data_quant.py \
  -m /home/q/deepseek-1.5b \
  -o /home/q/deepseek-1.5b/data_quant.json
```

Produces data_quant.json (21 samples). CPU inference takes about 20~40 minutes.

## Model Conversion (Key Step)

Key parameters of the conversion script (must be modified for RK3576; official example defaults to RK3588):

```python
# export_rkllm_rk3576.py (based on official export_rkllm.py, modified)
from rkllm.api import RKLLM
import os
os.environ['CUDA_VISIBLE_DEVICES']='-1'          # No GPU, force CPU

modelpath = '/home/q/deepseek-1.5b'
llm = RKLLM()
ret = llm.load_huggingface(model=modelpath, device='cpu',
                           dtype="float32", custom_config=None, load_weight=True)

dataset = "/home/q/deepseek-1.5b/data_quant.json"
target_platform = "RK3576"        # ★ Official example is RK3588, must change
quantized_dtype = "W4A16"         # ★ W4A16 halves memory, faster
quantized_algorithm = "grq"       # ★ grq algorithm pairs with W4A16
num_npu_core = 1                  # ★ RK3576 single core

ret = llm.build(do_quantization=True, optimization_level=1,
                quantized_dtype=quantized_dtype, quantized_algorithm=quantized_algorithm,
                target_platform=target_platform, num_npu_core=num_npu_core,
                dataset=dataset, hybrid_rate=0, max_context=4096)
ret = llm.export_rkllm(f"/home/q/deepseek-1.5b/deepseek-1.5b_W4A16_RK3576.rkllm")
```

Run:

```bash
/home/q/rkllm-venv/bin/python /home/q/deepseek-1.5b/export_rkllm_rk3576.py
```

Output: deepseek-1.5b_W4A16_RK3576.rkllm (~1.3GB). Takes about 75 minutes (CPU quantization).

## Cross-Compile the On-Board Demo

```bash
TC=<SDK>/prebuilts/gcc/linux-x86/aarch64/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu

# Copy deploy directory to workspace (avoid polluting the SDK repo)
cp -r <SDK>/external/rknn-llm/examples/DeepSeek-R1-Distill-Qwen-1.5B_Demo/deploy /home/q/llm_demo_deploy

# In CMakeLists.txt, change RKLLM_API_PATH to an absolute path:
#   set(RKLLM_API_PATH "/home/q/rkllm-runtime/Linux/librkllm_api")

mkdir -p /home/q/llm_demo_deploy/build && cd /home/q/llm_demo_deploy/build
cmake .. \
  -DCMAKE_SYSTEM_PROCESSOR=aarch64 -DCMAKE_SYSTEM_NAME=Linux \
  -DCMAKE_C_COMPILER=$TC-gcc -DCMAKE_CXX_COMPILER=$TC-g++ \
  -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON
make -j4 && make install
```

Output: /home/q/llm_demo_deploy/install/demo_Linux_aarch64/, containing llm_demo + lib/librkllmrt.so.

## Deploy to the Board

```bash
adb shell mkdir -p /root/llm_demo/lib
cd /home/q/llm_demo_deploy/install/demo_Linux_aarch64
adb push llm_demo /root/llm_demo/
adb push lib/librkllmrt.so /root/llm_demo/lib/
adb push /home/q/deepseek-1.5b/deepseek-1.5b_W4A16_RK3576.rkllm /root/llm_demo/
adb shell chmod +x /root/llm_demo/llm_demo
```

## NPU/CPU Frequency Locking (Required Before Running the Model)

The official fix_freq_rk3576.sh hardcodes NPU at 1GHz, but this board's NPU max is 950MHz — needs adaptation:

```bash
# On board (root)
echo userspace > /sys/class/devfreq/27700000.npu/governor
echo 950000000 > /sys/class/devfreq/27700000.npu/userspace/set_freq   # max 950MHz

# CPU big core (policy4) locked to max 2208MHz:
echo userspace > /sys/devices/system/cpu/cpufreq/policy4/scaling_governor
echo 2208000 > /sys/devices/system/cpu/cpufreq/policy4/scaling_setspeed
```

## Run

```bash
adb shell
cd /root/llm_demo
export LD_LIBRARY_PATH=./lib
sh fix_freq_rk3576_debian.sh
export RKLLM_LOG_LEVEL=1          # Print performance statistics
./llm_demo deepseek-1.5b_W4A16_RK3576.rkllm 2048 4096
# Parameters: 2048=inference batch size limit, 4096=max context length
```

After entering interactive mode, type 0 to test the chicken-rabbit problem, or enter a custom question.

# Measured Results

## Functional Verification

The chicken-rabbit problem (14 heads, 38 legs) is answered correctly: 9 chickens, 5 rabbits, with full thinking/reasoning process.

## Performance Data (NPU 950MHz Locked)

| **Metric** | **Measured Value** |
| --- | --- |
| Model load time | 1.54 s |
| Prefill | 103.31 tokens/s (40 tokens / 0.39s) |
| Generation speed | 10.29 tokens/s (459 tokens / 44.6s) |
| Peak memory | 1.10 GB (plenty of headroom in 4GB) |

## Quantization Scheme Comparison (RK3576 Official Benchmark, 1.5B)

| **Quantization** | **Memory Usage** | **Generation Speed** |
| --- | --- | --- |
| W8A8 | 1665 MB | 8.51 tokens/s |
| W4A16 | 932 MB | 14.45 tokens/s |

Conclusion: W4A16 is better for RK3576 (4GB RAM) — half the memory, ~70% faster.

# FAQ

| **Problem** | **Cause & Solution** |
| --- | --- |
| pip install rkllm-toolkit hangs | Wheel depends on large packages like torch 2.3.0; use --no-deps with system torch |
| rkllm.h: No such file or directory | After copying the deploy directory, CMakeLists relative path breaks; change to absolute path |
| aarch64-linux-gnu-g++: command not found | Use the full SDK prebuilts toolchain |
| Frequency lock to 1GHz fails | This board's NPU max is 950MHz; use echo 950000000 |
| failed to open model on board | Model file path wrong or not pushed |
| libgomp.so.1 not found | Install libgomp1 on the board |

# Future Extensions

- **OpenAI-compatible API service**: SDK includes examples/rkllm_server_demo, can start an HTTP service for remote calls

- **Firmware integration**: Package llm_demo + model into rootfs, auto-start on boot

- **Larger models**: 7B W4A16 ~4GB is tight for 4GB RAM, not recommended; 1.5B is the current best choice